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(54) Image processing apparatus, method and system 



(57) In order to enable plural users to share virtual 
environment wliich includes not only simple shape data 
but also data of an object of which shape Is highly com- 
plicated; a ternninal equipment displays the virtual envi- 
ronment obtained when viewed from the user of the ter- 
minal equipment on the basis of tree-structure data 
composing the shape data, light space data and the like 
as components, changes the tree-structure data when 
the shape data or the light space data is subjected to 
the process such as movement or the like, and transmits 
changed information of the tree-structure data to a serv- 
er: the server changes data in a data memory device 
based on the received changed information, and trans- 
mits the changed information to an another terminal 
equipment; and the another terminal equipment dis- 
plays the virtual environment obtained when viewed 
from the user of the another terminal equipment on the 
basis of the received changed information. 



FIG. 1 



102 



mm 

I — 



DATA 
MEMORY 
i DEVICE 



^01 






103 



1G4 



105 



a. 

lU 



Pumed by Jcuve '5001 PAWS <FR) 



BEST AVAILABLE COPY 



1 



EP 0 865 001 A2 



2 



Description 

BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates to image processing 
apparatus, method and system in which virtual environ- 
ment for virtual reality is generated and the generated 
virtual environment is shared by plural users. 

Related Background Art 

Conventionally, there has been provided a method 
to express three-dimensional object and space and pro- 
vide an observation image in arbitrary position and di- 
rection. 

In this method (1 ): the three-dinnensional object and 
space are expressed by using shape model data such 
as polygon data, curved surface data or the like, texture 
data representing surface attribute or pattern of the im- 
age, light source data and the like, and a view state (or 
appearance) of the space in the arbitrary position and 
direction is sketched on a display screen, whereby the 
three-dimensional space and object are reproduced. 

I n case of constructing three-dimensional virtual en- 
vironment by this method, there has been a method to 
express elements (coordinate conversion data= shape 
data, surface attribute data, illumination and the like) 
composing such the virtual environment by means of 
tree structure. This is because the space, the ground, 
buildings, rooms, furniture, illuminators, ornaments and 
the like composing the three-dimensional space are 
considered to be essentially in hierarchical nested rela- 
tion. For example, since the ornament on the desk is 
moved according to arrangement of the desk, the orna- 
ment is in the relation depending on the arrangement of 
the desk. Therefore, often it is better convenient for the 
ornament to be relatively arranged based on a coordi- 
nate system in which the desk exists, thereby applying 
the data structure in which the arrangements of such the 
elements are in the hierarchically dependent relation. As 
the method to do so, the virtual environment is ex- 
pressed by n-tree structure. 

For example, Fig. 42 shows an example of one sim- 
ple virtual environment. In this case, if it pays attention 
to a room, a desk and a sofa in the space, the room is 
described on a coordinate system C2 converted from a 
world coordinate system Cq through a coordinate con- 
version T2: and the desk and the sofa in this room are 
described respectively on coordinate systems C3 and 
C4 respectively converted from the coordinate system 
C2 tlirough coordinate conversions T3 and T4. Further 
a pot on the desk is described on a coordinate system 
C5 converted from the coordinate system C3 through a 
coordinate conversion T5. Fig. 43 shows tree structure 
schematically expressing such relation. 

Further, there has been a method (2) that the light 



space data is generated based on an image group ob- 
tained by real-photographing the three-dimensional ob- 
ject and space, the Image visible in arbitrary position and 
direction is formed based on the generated light space 

s data, and the formed image is displayed to reproduce 
the three-dimensional object and space. In this method, 
a shape of an object is not obtained, but the object is 
reproduced as a set (or gather) of light. 

Furthermore, there has been provided a method to 

10 perform cooperative working by plural users. 

In this method (3), a data memory device storing 
data common to the plural users and a sen/er device for 
managing processing on the data memory device are 
provided, and each user communicates with the server 

15 device to refer to and change the data stored in the data 
memory device or each user directly communicates with 
other user with respect to the change of the data in the 
data memory device, thereby maintaining the data of 
each user in the same state. 

20 Furthermore, there has been provided a method (4) 
that, by utilizing the method (3) to perform the coopera- 
tive working and the method (1) to express the virtual 
environment, the state of the virtual environment con- 
sisting of geometrical shape data is maintained to be 

25 identical, whereby the virtual environment is shared by 
the plural users. 

However, in the above-described conventional 
methods (1) to (4), there are following problems. 

In the method (1), it is difficult to generate or repro- 

30 duce the shape data for the object of which shape is 
highly complicated. Further, it is also difficult to capture, 
from a real object, the shape data of this object of which 
shape is complicated by using a three-dimensional 
measurement device. Especially, it is more difficult to 

35 reproduce the real object having the already-existent 
complicated shape or surface pattern or a complicated 
reflection characteristic (i.e., absorption transmission 
characteristic). In addition, generally^ it is easy to ex- 
press an artificial object but difficult to express a natural 

40 object. However, in this method, there is an advantage 
that the artificial and simple three-dimensional space 
such as the room or a row of houses (or street) which is 
mainly composed of planes can be expressed in a small 
data amount. Further the method to express the virtual 

•^5 environment by using the tree structure is an excellent 
method. 

In the method (2), when the light space data has 
been generated from a photographed image of a certain 
amount, then an observation image at an arbitrary vlew- 

50 point position (although including restricted condition) 
can be formed. Since the shape of the object is not re- 
stored and the object is described based on the photo- 
graphed image of the object itself, this method is an ex- 
cellent method which can express the object in a highly 

55 realistic manner However since the large data amount 
is necessary and the photographing becomes difficult 
when the object has a wide range, this method is suita- 
ble to describe the three-dimensional object rather than 
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the wide space. That is. this method is not suitable to 
express the three-dimensional space such as the room 
or the row of the houses. 

Further the above conventional methods (1) and 
(2) have the different features respectively. However in s 
the conventional method (4), the method or technique 
which is combined with the method (3) to realize the 
common virtual space is only the method (1) of which 
data amount is small. That is, the method (2) could not 
be used in the method (4) because of its large data io 
amount. For this reason, the common virtual environ- 
ment realized by the method (4) comes to include the 
feature in the method (1)= whereby such the common 
virtual environment becomes the virtual environment 
which can not include the object such as the natural ob- '5 
ject of which shape is highly complicated. 

SUMMARY OF THE INVENTION 

An object of the present invention is to enable plural 20 
users to share virtual environment including simple 
shape data and highly complicated shape data. 

An image processing apparatus according to the 
present invention comprises: 

25 

a tree-structure data memory means for storing 
tree-structure data, the tree-structure data including 
as components shape data geometrically describ- 
ing a three-dimensional space and a three-dimen- 
sional object and light space data describing the 
three-dimensional space and the three-dimension- 
al object based on a real-photographed image; 
a displaying means for.displaying a virtual three-di- 
mensional space in which the shape data and the 
light space data are mixed with each other, based 3S 
on the tree-structure data stored in the tree-struc- 
ture data memory means: 

a processing means for processing at least one of 
the shape data and the light space data both dis- 
played; 

a changing means for changing the tree-structure 
data stored in the tree-structure data memory 
means, based on the processing content by the 

processing means: 

a re-displaying means for re-displaying the virtual ^5 
three-dimensional space, based on the data chang- 
ing content by the changing means: 
an indication information generating means for gen- 
erating indication information indicating the chang- 
ing content of the tree-structure data, based on the 50 
processing content by the processing means: and 
a transmitting means for transmitting the indication 
information generated by the indication information 
generating means to an another information 
processing apparatus connected to the image 
processing apparatus. 

' Further an image processing apparatus according 



to the present invention comprises: 

a tree-structure data memory means for storing 
tree-structure data, the tree-structure data including 
as components shape data geometrically describ- 
ing a three-dimensional space and a three-dimen- 
sional object and light space data describing the 
three-dimensional space and the three-dimension- 
al object based on a real-photographed image; 
a displaying means for displaying a virtual three-di- 
mensional space in which the shape data and the 
light space data are mixed with each other, based 
on the tree-structure data stored in the tree-struc- 
ture data memory means: 

a receiving means for receiving indication informa- 
tion indicating a changing content of the tree-struc- 
ture data, transmitted from an another information 
processing apparatus connected to the image 
processing apparatus; 

a changing means for changing the tree-structure 
data stored in the tree-structure data memory 
means, based on the indication information re- 
ceived by the receiving means; and 
a re-displaying means for re-displaying the virtual 
three-dimensional space, based on the data chang- 
ing content by the changing means. 

Furthermore, an image processing method accord- 
ing to the present invention which is used in an image 
processing apparatus comprising a tree-structure data 
memory means for storing tree-structure data, the tree- 
structure data including as components shape data ge- 
ometrically describing a three-dimensional space and a 
three-dimensional object and light space data describ- 
ing the three-dimensional space and the three-dimen- 
sional object based on a real-photographed image, the 
method comprises: 

a displaying step of displaying a virtual three-dimen- 
sional space in which the shape data and the light 
space data are mixed with each other based on the 
tree-structure data stored in the tree-structure data 
memory means: 

a processing step of processing at least one of the 
shape data and the light space data both displayed; 
a changing step of changing the tree-structure data 
stored in the tree-structure data memory means, 
based on the processing content in the processing 
step: 

a re-displaying step of re-displaying the virtual 
three-dimensional space, based on the data chang- 
ing content in the changing step: 
an indication information generating step of gener- 
ating indication information indicating the changing 
content of the tree-structure data, based on the 
processing content in the processing step: and 
a transmitting step of transmitting the indication in- 
formation generated in the indication information 
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generating step to an another information process- 
ing apparatus connected to the image processing 
apparatus. 

Furthermore, an image processing method accord- 5 
ing to the present invention which is used in an image 
processing apparatus comprising a tree-structure data 
memory means for storing tree-structure data, the tree- 
structure data including as connponents shape data ge- 
ometrically describing a three-dimensional space and a 
three-dimensional object and light space data describ- 
ing the three-dimensional space and the three-dimen- 
sional object based on a real -photographed image, the 
method comprises: 

15 

a displaying step of displaying a virtual three-dimen- 
sional space in which the shape data and the light 
space data are mixed with each other, based on the 
tree-structure data stored in the tree-structure data 
memory means: 

a receiving step of receiving indication information 
indicating a changing content of the tree-structure 
data, transmitted from an another information 
processing apparatus connected to the image 
processing apparatus: 

a changing step of changing the tree-structure data 
stored in the tree-structure data memory means, 
based on the indication information received in the 
receiving step: and 

a re-displaying step of re-displaying the virtual 
three-dimensional space, based on the data chang- 
ing content in the changing step. 

Furthermore, a computer-readable memory medi- 
um according to the present invention which is used in -35 
an image processing apparatus comprising a tree-struc- 
ture data memory means for storing tree-structure data, 
the tree-structure data including as components shape 
data geometrically describing a three-dimensional 
space and a three-dimensional object and light space -^o 
data describing the three-dimensional space and the 
three-dimensional object based on a real-photographed 
image, and which stores a computer program including: 

a displaying module for displaying a virtual three- -^5 
dimensional space in which the shape data and the 
light space data are mixed with each other based 
on the tree-structure data stored in the tree-struc- 
ture data memory means: 

a processing module for processing at least one of so 
the shape data and the light space data both dis- 
played: 

a changing module for changing the tree-structure 
data stored in the tree-structure data memory 
means, based on the processing content in the 55 
processing step: 

a re-displaying module for re-displaying the virtual 
three-dimensional space, based on the data chang- 



ing content: 

an indication information generating module for 
generating indication information indicating the 
changing content of the tree-structure data^ based 
on the processing content; and 
a transmitting module for transmitting the generated 
indication information to an another information 
processing apparatus connected to the image 
processing apparatus. 

Furthermore, a computer-readable memory medi- 
um according to the present invention which is used in 
an image processing apparatus comprising a tree-struc- 
ture data memory means for storing tree-structure data, 
the tree-structure data including as components shape 
data geometrically describing a three-dimensional 
space and a three-dimensional object and light space 
data describing the three-dimensional space and the 
three-dimensional object based on a real-photographed 
image, and which stores a computer program including: 

a displaying module for displaying a virtual three- 
dimensional space in which the shape data and the 
light space data are mixed with each other, based 
on the tree-structure data stored in the tree-struc- 
ture data memory means; 

a receiving module for receiving indication informa- 
tion indicating a changing content of the tree-struc- 
ture data, transmitted from an another information 
processing apparatus connected to the image 
processing apparatus: 

a changing module for changing the tree-structure 
data stored in the tree-structure data memory 
means, based on the received indication informa- 
tion; and 

a re-displaying module for re-displaying the virtual 
three-dimensional space, based on the data chang- 
ing content. 

Furthermore, an image processing system accord- 
ing to the present invention in which a first terminal 
equipment is connected to a second terminal equipment 
through a server comprises: 

the first terminal equipment further comprising, 

a first tree-structure data memory means for 
storing tree-structure data, the tree-structure 
data including as components shape data ge- 
ometrically describing a three-dimensional 
space and a three-dimensional object and light 
space data describing the three-dimensional 
space and the three-dimensional object based 
on a real-photographed image, 
a first displaying means for displaying a virtual 
three-dimensional space in which the shape 
data and the light space data are mixed with 
each other, based on the tree-structure data 



4 



7 



EP 0 865 001 A2 



stored In the first tree-structure data memory 
means, 

a processing means for processing at least one 
of the shape data and the light space data both 
displayed, ^ 
a first changing means for changing the tree- 
structure data stored in the first tree-structure 
data memory means, based on the processing 
content by the processing means, 
a first re-displaying means for re-displaying the io 
virtual three-dimensional space, based on the 
data changing content by the first changing 
means.. 

an indication information generating means for 
generating indication information indicating the ^5 
changing content of the tree-structure data, 
based on the processing content by the 
processing means, and 
a first transmitting means for transmitting the 
indication information generated by the indica- 20 
tion information generating means to the server 
connected to the first terminal equipment: 

the server further comprising, 

25 

a first receiving means for receiving the indica- 
tion information transmitted f ronn the first termi- 
nal equipment, and 

a second transmitting means for transmitting 
the indication information received by the first ^0 
receiving means to the second terminal equip- 
ment; and 

the second terminal equipment further comprising, 

35 

a second tree-structure data memory means 
for storing tree-structure data, the tree-struc- 
ture data including as components shape data 
geometrically describing a three-dimensional 
space and a three-dimensional object and light -^o 
space data describing the three-dimensional 
space and the three-dimensional object based 
on a real -photographed image, 
a second displaying means for displaying a vir- 
tual three-dimensional space in which the -^5 
shape data and the light space data are mixed 
with each other, based on the tree-structure da- 
ta stored in the second tree-structure data 
memory means, 

a second receiving means for receiving indica- so 
tion Information indicating a changing content 
of the tree-structure data, transmitted from the 
server connected to the second terminal equip- 
ment, 

a second changing means for changing the 55 
tree-structure data stored in the second tree- 
structure data memory means, based on the in- 
dication information received by the second re- 



ceiving means, and 

a second re-displaying means for re-displaying 
the virtual three-dimensional space, based on 
the data changing content by the second 
changing means. 

Furthermore, an image processing system accord- 
ing to the present invention in which a first terminal 
equipment and a second terminal equipment are con- 
nected to each other, comprises: 

the first terminal equipment further comprising, 

a first tree-structure data memory means for 
storing tree-structure data, the tree-structure 
data including as components shape data ge- 
ometrically describing a three-dimensional 
space and a three-dimensional object and tight 
space data describing the three-dimensional 
space and the three-dimensional object based 
on a real-photographed image, 
a first displaying means lor displaying a virtual 
three-dimensional space in which the shape 
data and the light space data are mixed with 
each other based on the tree-structure data 
stored in the first tree-structure data memory 
means, 

a processing means for processing at least one 
of the shape data and the light space data both 

displayed: 

a first changing means for changing the tree- 
structure data stored in the first tree-structure 
data memory means, based on the processing 
content by the processing means, 
a first re-displaying means for re-displaying the 
virtual three-dimensional space, based on the 
data changing content by the first changing 
means. 

an indication information generating means for 
generating indication information indicating the 
changing content of the tree-structure data, 
based on the processing content by the 
processing means, and 

a transmitting means for transmitting the indi- 
cation information generated by the indication 
information generating means to the second 
terminal equipment connected to the first termi- 
nal equipment: and 

the second terminal equipment further comprising, 

a second tree-structure data memory means 
for storing tree-structure data, the tree-struc- 
ture data including as components shape data 
geometrically describing a three-dimensional 
space and a three-dimensional object and light 
space data describing the three-dimensional 
space and the three-dimensional object based 
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on a real-photographed image, 
a second displaying nneans for displaying a vir- 
tual three-dimensional space in which the 
shape data and the light space data are mixed 
with each other based on the tree-structure da- 5 
ta stored in the second tree-structure data 
memory means, 

a receiving means for receiving indication infor- 
mation indicating a changing content of the 
tree-structure data, transmitted from the first 
terminal equipment connected to the second 

terminal equipment, 

a second changing means for changing the 
tree-structure data stored in the second tree- 
structure data memory means, based on the in- ^5 
dication information received by the receiving 
means, and 

a second re-displaying means for re-displaying 
the virtual three-dimensional space, based on 
the data changing content by the second 20 
changing means. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram showing a first embodi- 2S 
ment: 

Fig 2 is a view showing the structure of a data mem- 
ory device 101 included in a server device 102: 
Fig. 3 is a block diagram showing the structure of 
the server device 102: 

Fig. 4 is a view showing the structure of a command 
buffer 305; 

Fig. 5 is a view showing the structure of a terminal 
equipment; 

Fig. 6 is a view showing the structure of a data mem- 3S 
ory device 503 included in the terminal equipment: 
Fig. 7 is a flow chart showing processing by the 
server device 102 in the first embodiment: 
Fig. 8 is a flow chart showing the processing by the 
server device 1 02 when a user record addition com- 
mand is received: 

Fig. 9 is a flow chart showing the processing by the 
server device 1 02 when a user record deletion com- 
mand is received: 

Fig. 10 is a flow chart showing the processing by 
the server device 102 when a user record attnbute 
inquiry command is received: 
Fig. 11 is a flow chart showing processing by the 
server device 1 02 when a user record change com- 
mand is received: 

Fig. 12 is a flow chart showing the processing by 

^he server device 102 when a maximum user record 

value inquiry command is received; 

Fig. 13 is a flow chart showing the processing by 

the server device 102 when an object record addi- ^5 

tion command is received: 

Fig. 14 is a flow chart showing the processing by 
the server device 102 when an object record dele- 



tion command is received: 

Fig. 15 is a flow chart showing the processing by 
the server device 102 when an object record at- 
tribute inquiry command is received; 
Fig. 16 is a flow chart showing the processing by 
the server device 102 when an object record 
change command is received: 
Fig. 17 is a flow chart showing the processing by 
the seo/er device 1 02 when an object record lock 
command is received; 

Fig. 18 is a flow chart showing the processing by 
the server device 102 when an object record unlock 

command is received; 

Fig. 19 is a flow chart showing the processing by 
the server device 102 when a maximum object 
record value inquiry command is received; 
Fig. 20 is a flow chart showing the processing by 
the terminal equipment in the first embodiment; 
Fig. 21 is a flow chart showing initialization process- 
ing by the terminal equipment; 
Fig. 22 is a flow chart showing object record update 
processing by the terminal equipment: 
Fig. 23 is a flow chart showing user record update 
processing by the terminal equipment: 
Fig. 24 is a flow chart showing sketch processing 
by the terminal equipment: 

Fig. 25 is a view showing an example of virtual en- 
vironment generated in the first embodiment; 
Fig. 26 is a view showing the virtual environment in 
Fig. 25 by using tree structure: 
Fig. 27 is a view showing text data expressing the 
tree structure in Fig. 26; 

Fig. 28 is a flow chart showing processing to present 

the tree structure to an obsen/er (user); 

Fig. 29 is a view showing a sketch example of the 

virtual environment which is being observed by an 

observer 1 in the first embodiment: 

Fig. 30 is a view showing a sketch example of the 

virtual environment which is being observed by an 

observer 2 in the first embodiment: 

Fig. 31 is a view showing a sketch example of the 

virtual environment which is being observed by the 

observer 1 in the first embodiment: 

Fig. 32 is a view showing a sketch example of the 

virtual environment which is being observed by the 

observer 2 in the first embodiment: 

Fig. 33 is a flow chart showing user operation 

processing at the terminal equipment: 

Figs. 34A and 34B are schematic views showing a 

three-dimensional space to explain a principle of a 

light space: 

Fig. 35 is a view showing relation between one point 
in a real space and light passing through such the 
point: 

Fig. 36 is a view showing mapping that the light 
passing through one point in the real space is 
mapped to an x-u space being the light space: 
Figs. 37A and 37B are views showing a principle of 
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mutual conversion between the real space and the 
light space: 

Fig. 38 is a view showing a principle that an image 
Is fornned from a nnulti-viewpoint image used in a 
second embodiment; 

Fig. 39 is a flow chart showing processing that the 
image is formed from the multi-viewpoint image 
used in the second embodiment; 
Fig. 40 is a view showing a principle of distortion 
correction in a longitudinal direction in the image 
forming of the second embodiment; 
Fig. 41 is a block diagram showing a third embodi- 
ment; 

Fig. 42 is a view showing an example of virtual en- 
vironment in the prior art (1); and 
Fig. 43 is a view showing the virtual environment in 
Fig. 42 by using tree structure. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

(First Embodiment) 

Fig. 1 is a block diagram showing a shared virtual 
environment generating system in the first embodiment. 

Reference numeral 101 denotes a data memory de- 
vice which stores data representing a state of shared 
virtual environment, and reference numeral 102 de- 
notes a server device which manages changing of the 
data representing the state of the shared virtual environ- 
ment. Contents of the data memory device 1 01 can be 
changed only responsive to an instruction from the serv- 
er device 102. Reference numerals 103 to 105 denote 
terminal equipments each having an interface device. 
In the first embodiment, the interface device displays the 
state of the shared virtual environment to users, and al- 
so transmits to the server device 102 an instruction to 
change the data representing the state of the shared vir- 
tual environment. 

In the first embodiment, it is assumed that the data 
memory device 101 is provided within the server device 
1 02. However: the data memory device 101 may be pro- 
vided anywhere on a network to which the server device 
102 and the terminal equipments 103 to 105 are con- 
nected 

Fig. 2 shows the data structure in the data memory 
device 101 . 

Reference numeral 201 denotes information repre- 
senting an initial tree state. As the information 201 , data 
representing an initial state of the shared virtual envi- 
ronment Is stored when the server device 102 is initiat- 
ed. Reference numerals 202 to 204 denote object 
records respectively including information of objects ex- 
isting in the shared virtual environment. The object 
record 202 is composed of an object number 202a rep- 
resenting a number of the object, a lock user number 
202b representing a number of the user who is locking 
the object, and an attribute filed 202c including various 



data concerning the object. 

Concretely the attribute filed 202c includes position da- 
ta 202c1 representing position and direction of the ob- 
ject in a world coordinate system or position and direc- 

5 tion of the object in the tree of the shared virtual envi- 
ronment, shape data 202c2 representing a geometrical 
shape of the object, surface attribute data 202c3 repre- 
senting a surface reflection coefficient of the object or 
the like, and light space data 202c4 of the object. Simi- 

10 larly. each of the object records 203 and 204 is com- 
posed of an object number a lock user number and an 
attribute field. 

Reference numeral 205 denotes a maximum object 
record value. As the value 205, the maximum object 

15 number in the object records existing within the data 
memory device 101 is stored. 

Reference numerals 206 to 208 denote user 
records which respectively include information of the us- 
ers common to the virtual environment. The user record 

20 206 is composed of a user number 206a representing a 
number of the user a latest object number 206b repre- 
senting the latest object of which state is held by the 
user, a latest user number 206c representing the latest 
user holding the state, and attribute data 206d repre- 

25 senting the state of position, direction and the like of a 
user's viewpoint. Similarly, each of the user records 207 
and 208 is composed of a user number a latest object 
number a latest user number and an attribute data. 
Reference numeral 209 denotes a maximum user 

30 record value. As the value 209= the maximum user 
number in the user records existing in the data memory 
device 101 is stored. 

Fig. 3 shows the internal structure of the server de- 
vice 102. 

35 Reference numeral 303 denotes a CPU which exe- 
cutes processing procedure, and reference numeral 
304 denotes a processing program memory device 
which stores the processing procedure. Reference nu- 
meral 305 denotes a first-in first-out command buffer 

•^0 which temporarily stores a command transmitted from 
the terminal equipment. Reference numeral 306 de- 
notes an interface which is used to receive the com- 
mand from the terminal equipment and transmit the data 
to the terminal equipment. Further, reference numeral 

-^5 307 denotes a bus which connects the above units to 
others 

Fig. 4 shows the internal structure of the command 
buffer 305. 

The commands transmitted from plural communi- 
50 cation terminals are sequentially stored in the command 
buffer 305. The CPU 303 fetches the command from the 
leading position of the buffer one by one and then proc- 
esses the fetched command. Reference numerals 401 , 
402 and 403 denote commands transmitted respective- 
55 ly from the terminal equipments. 

The structure of the command 401 will be ex- 
plained. A user number 401 a represents a user number 
of the terminal equipment which transmitted the com- 
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mand. A return address 401b represents an address to 
be used as a transnnission destination when the server 
device 102 replies to the command. A command 
number 401c represents a kind of the command. A user 
record 401 d represents the user record which is referred 
to or changed by the command 401 . Further, an object 
record 401 e represents the object record which is re- 
ferred to or changed by the command 401 . Since all the 
data are not always necessary for all the kinds of the 
commands, unnecessary data can be omitted or can- 
celled. 

Fig. 5 shows the internal structure of the terminal 
equipment 103. It should be noted that the internal struc- 
tures of the terminal equipments 104 and 105 are iden- 
tical with that of the terminal equipment 103. 

Reference numeral 501 denotes a CPU which exe- 
cutes processing procedure on the side of the terminal 
equipment in the first embodiment. Reference numeral 
502 denotes a processing program memory device 
which stores the processing procedure. Reference nu- 
meral 503 denotes a data memory device which holds 
the data representing the state of the shared virtual en- 
vironment. The data is shared by the data memory de- 
vice 503 and the data memory device 101 on the side 
of the server device 102. Reference numeral 504 de- 
notes a frame buffer which stores images representing 
the course of processing and the processed results such 
that these images can be sent to a display device. Ref- 
erence numeral 505 denotes a window system which 
acts as the display device to display the images in the 
frame buffer 504. Reference numeral 506 denotes a 
keyboard which inputs characters and instructions in the 
course of processing. Reference numeral 507 denotes 
a mouse which is used to designate a position on a dis- 
play screen. Reference numeral 508 denotes an inter- 
face device which is used to transmit the command to 
the server device 102 and receive the data from the 
server device 102. Further reference numeral 509 de- 
notes a bus which connects the above units to others. 

Fig. 6 shows the data structure of the data memory 
device 503. 

Reference numerals 601 and 603 to 610 respec- 
tively denote the same elements as the corresponding 
elements in the data structure of the data memory de- 
vice 101 shown in Fig 2. The content of the data mem- 
ory device 503 in each terminal equipment is maintained 
to be substantially identical with that of the data memory 
device 101 on the side of the server device 102 through 
communication between the server device 102 and the 
terminal equipment. As a user number 602: a number 
which is given by the server device 102 at the time of 
initializing the terminal equipment is stored. 

Hereinafter, the processing on the side of the server 
device 1 02 in the first embodiment will be explained with 
reference to flow charts shown in Figs. 7 to 19. The 
processing program for such the processing has been 
stored in the processing program memory device 304, 
and is executed by the CPU 303. 



Fig. 7 is the flow chart showing the processing by 
the server device 1 02 in the first embodiment. 

Initially, the data memory device 101 is initialized in 
a step S701. Concretely, after deleting all the object 
5 records and the user records, the information 201 rep- 
resenting the initial tree state is read. In the information 
201 representing the initial tree state, geometrical shape 
data having the surface attribute data, or light space da- 
ta is included. This information 201 can be previously 
10 generated by using some device such as an already- 
existent modeler or the like. 

As the shape data, there are polygon data such as 
a triangular patch, free curved surface data such as 
NURBS, or the like. The surface attribute data repre- 
ss sents a material, a reflection characteristic, a pattern 
and the like of the shape. The pattern is input as texture 
data. The texture data represents the pattern on the sur- 
face of the polygon data, the free curved surface data 
or the like, and describes positional relation between the 
20 pattern image data and the shape being the object to 
which such the pattern or image is pasted. 

If a plane of z = 0 in a coordinate system shown in 
Figs. 34A and 34B is assumed, the light space data ex- 
presses the three-dimensional space as a set (or gath- 
25 er) of light passing through this plane (referred as stand- 
ard plane 34 hereinafter). 

In the method which uses this light space data, the 
image which can be observed at a viewpoint position P 
on a point z in the three-dimensional space is equivalent 
30 to the image which can be obtained by sampling only 
the light passing through the position P from the set of 
the light passing through the standard plane 34 (see Fig. 
35). Generally each light is represented by a position 
(x,y) on the standard plane 34 through which the light 
35 passes, an angle 0 between the light and an x axis, an 
angle \\f between the light and a y axis, a time t when 
the light passes through the plane and a color (r.g.b) of 
the light. However, because of a problem that a calcu- 
lating amount and a data amount becomes huge, actu- 
40 ally it is often assumed that the object is a still object 
and there is no parallax in a y-axis direction. Under this 
assumption, u = tan<|) is set and each light is projected 
on an x-u space. In a case where a light space is handled 
by using the x-u space on which the light was projected, 
45 the light passing through one point (viewpoint position) 
P has a liner locus as shown in Fig. 36. This locus is 
expressed by following equations. 



u = tan(t> (2) 

55 Here, reference symbol (X,Z) denotes an observa- 
tion viewpoint position, reference symbol x denotes a 
position at which the light crosses the x axis in the x-u 
space, and reference symbol 0 denotes the angle be- 
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tween the light and the z axis. 

Initially, as shown in Figs. 37A and 37B. the linear 
loci are obtained from the image obtained at numerous 
viewpoint positions, and it is assumed that the x-u space 
is densely filled with a group of these loci. At this time, 
as shown in Figs. 37A and 37B. if the locus passing 
through the position Q in the x-u space is obtained and 
the color of the light already recorded on this locus is 
inversely obtained, the image at a viewpoint position Q 
on the point z can be obtained. 

Subsequently, the processing after terminating the 
initializing of the server device 102 in the step S701 will 
be explained. The server device 102 enters into a loop 
to wait for the command from the terminal equipment. 
Then, it is confirmed in a step S703 whether or not the 
command has been inputted in the command buffer 305 
from the terminal equipment. If the command buffer 305 
is empty, the server device 102 again enters into the 
loop to wait for the command from the terminal equip- 
ment. On the other hand, if the command buffer 305 is 
not empty, the flow advances to a step S704 to operate 
the data memory device 101 according to the command 
at the leading position in the buffer 305. As a result of 
the operation, when it becomes necessary to transmit 
the data to the terminal equipment, the data is transmit- 
ted through the interface 306. At this time, the data is 
transmitted to all the terminal equipments connected to 
the server device 102 such that all the terminal equip- 
ments have the identical data. After terminating the 
processing in the step S704, the server device 102 
again enters into the loop to wait for the command from 
the terminal equipment. 

There are 12 kinds of commands read from the 
command buffer 305 In the step S704, i.e., a user record 
addition command, a user record deletion command, a 
user record attribute inquiry command, a user record 
change command, a maximum user record value inquiry 
command, an object record addition command, an ob- 
ject record deletion command, an object record attribute 
inquiry command, an object record change command, 
an object record lock command, an object record unlock 
command, and a maximum object record value inquiry 
command. Figs. 8 to 19 are the flow charts respectively 
showing the processing by the server device 102 at the 
respective times when the above 1 2 kinds of the com- 
mands are received. 

Fig. 8 is the flow chart showing the processing by 
the sen/er device 102 at the time when the user record 
addition command Is received. In a step S801 , the max- 
imum user record value 209 In the data memory device 
101 is increased by one. Then, In a step S802, a new 
user record is formed In the data memory device 101. 
In this case, as the user number of the new record, the 
maximum user record value 209 is stored. Finally in a 
step S803, the information 201 representing the maxi- 
mum user record value 209 and the initial tree stale is 
returned. 

' Fig. 9 is the flow chart showing the processing by 



the server device 102 at the time when the user record 
deletion command is received. When the user record 
deletion command is. received from a user U, in a step 

5901 . lock user numbers of all the user records included 
5 In the data memory device 101 are checked, and the 

lock user number of the user record which is being 
locked by the user U Is set to be "0". Finally in a step 

5902, the user record corresponding to the user U is 
deleted from the data record device 101 . 

10 Fig. 10 is the flow chart showing the processing by 
the server device 102 at the time when the user record 
attribute inquiry command is received. When the at- 
tribute Inquiry command of a user record R is received 
from the user U, it is confirmed in a step S1001 whether 

IS or not the user record R exists in the data memory de- 
vice 101 . If exists, the user record R is returned in a step 
Si 002. On the other hand, if not exist, the data repre- 
senting "deletion" is returned in a step SI 003. When the 
processing in the step SI 002 or SI 003 terminates, the 

20 latest user number of the user record R Is checked in a 
step 81004. If the checked number is smaller than R, it 
is replaced by R. 

Fig. 11 is the flow chart showing the processing by 
the server device 102 at the time when the user record 

25 change command is received. When the user record 
change command is received from the user 11= the at- 
tribute filed of the user record in the command is set to 
be P. Then, In a step Si 101 , the attribute field of the user 
record R In the data memory device 101 is replaced by P. 

30 Fig. 1 2 is the flow chart showing the processing by 
the server device 102 at the time when the maximum 
userrecord value inquiry command is received. In a step 
SI 201 , the maximum user record value 209 in the data 
memory device 101 Is returned. 

35 Fig. 1 3 Is the flow chart showing the processing by 
the server device 1 02 at the time when the object record 
addition command is received. In a step 81 301, the 
maximum object record value 205 in the data memory 
device 101 is increased by one. Then, in a step SI 302, 

40 a new object record is formed in the data memory device 
1 01 . At this time, the object number is set to be the max- 
imum object record value 205, and the attribute is set to 
be that of the object record included In the command. 
Fig. 1 4 is the flow chart showing the processing by 

45 the server device 1 02 at the time when the object record 
deletion command is received. When the deletion com- 
mand of the object record R is received from the user 
U, it is checked in a step SI 401 whether or not the object 
record R in the data memory device 1 01 is being locked 

so by the user U. If not locked, no processing Is performed. 
On the other hand, if locked. R Is deleted in a step 
S1402. 

Fig. 15 is the flowchart showing the processing by 
the server device 1 02 at the time when the object record 
55 attribute inquiry command Is received. When the at- 
tribute inquiry command of the object record R is re- 
ceived from the user U, it is confirmed In a step SI 501 
whether or not R exists In the data memory device 101 . 
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If exists, R is returned in a step Si 502. On the other 
hand, if not exit, the data representing "deletion" is re- 
turned in a step SI 503. When the processing in the step 
S 1 502 or S 1 503 terminates, the latest object number of 
the user record U Is checked in a step 81504. If the ob- 
ject number is smaller than R, it is replaced by R. 

Fig. 16 is the flow chart showing the processing by 
the server device 102 at the time when the object record 
change command is received. When the object record 
change command is received from the user U. it is 
checked in a step SI 601 whether or not the object 
record R of the data memory device 1 01 is being locked 
by the user U. If not locked, no processing is performed. 
On the other hand, if locked, the attribute field of the 
object record in the command is set to be P. Then, in a 
step SI 602. the attribute field of the user record R in the 
data memory device 101 is replaced by R 

Fig. 17 is the flow chart showing the processing by 
the server device 1 02 at the time when the object record 
lock command is received. When the lock command of 
the object record R is received from the user it is 
checked in a step SI 701 whether or not the object 
record R of the data memory device 101 is "0". If not "0", 
no processing is performed. On the other hand, if "0", 
the lock user number of the object record R is rewritten 
intoU in a step SI 702. 

Fig. 18 is the flow chart showing the processing by 
the server device 1 02 at the time when the object record 
unlock command is received. When the lock command 
of the object record R is received from the user U. it is 
checked in a step Si 801 whether or not the object 
record R of the data memory device 101 Is U. If not U, 
no processing is performed. On the other hand, if U. the 
lock user number is rewritten into "0" in a step SI 802. 

Fig. 19 is the flow chart showing the processing by 
the server device 102 at the time when the maximum 
object record value inquiry command is received. In a 
step Si 901. the maximum object record value 205 of 
the data memory device 101 is returned. 

The server device 102 repeats the processing on 
the above-described 12 kinds of the commands as far 
as the command is stored in the command buffer 305. 
If the data memory device 101 of the server device 102 
is rewritten, identical rewriting information is sent from 
the server device 102 to the data memory devices 503 
of all the terminal equipments. 

Subsequently, the processing on the side of theter- 
nninal equipment 103 in the first embodiment will be ex- 
plained hereinafter with reference to flow charts shown 
in Figs. 20 to 24, 28 and 33. The programs for the 
processing have been stored in the processing progrann 
memory device 502 and are executed by the CPU 501 . 

Fig. 20 is the flow chart showing the processing by 
the terminal equipment 103 in the first embodiment. 

Initially, the data memory device 503 is initialized in 
a step S2001 , and the flow enters into a sketch loop. In 
the sketch loop, the object record of the data memory 
device 503 is updated in a step S2002. Then, in a step 



52003, the user record of the data memory device 503 
is updated. After terminating the updating, in a step 

52004, sketch processing is performed on the frame 
buffer 504 by using the data in the data memory device 

5 503. After terminating the sketch processing, in a step 

52005, a user's instruction is received from the key- 
board 506 or the mouse 507. In a case where the user 
instructs to terminate the processing, end processing is 
performed in a step S2006- In other cases, the com- 

10 mand to change the content of the data memory device 
101 is issued to the server device 102 according to the 
user's instruction, and the flow again enters into the 
sketch loop and returns to the step S2O02. 

Fig. 21 is the flow chart showing the detailed 

IS processing in the initializing step S2001 . 

In a step S2101 , all the object records and the user 
records are deleted. Then, in a step S2102. the user 
record addition command is issued to the server device 
102. In the command, a return address is set as the ter- 

20 minal equipment 103. In a step S2103, when the infor- 
mation 201 representing the initial tree state and the us- 
er number are returned from the server device 102, the 
data memory device 503 records them respectively as 
the information 601 representing the initial tree state and 

2S the user number 602. 

Fig. 22 is the flow chart showing the detailed 
processing in the step S2002 to update the object record 
of the data memory device 503. 

Initially, in a step 8220 1 , the maximum object record 

30 value inquiry command is issued. In the command, a 
return address is set as the address of the terminal 
equipment 103. Then, the maximum object record value 
205 is received from the server device 102 in a step 
S2202. and the received value is compared with the 

35 maximum object record value 606 of the data memory 
device 503 in a step 82203. If the two values are equal 
to each other, the flow advances to a step S2205 On 
the other hand, if not equal, the flow advances to a step 
S2204. 

40 In the step S2204. the new object records are 
formed by a difference between the maximum object 
record value 606 of the data memory device 503 and 
the maximum object record value 205 received from the 
server device 102. and the maximum object record val- 

45 ue 606 of the data memory device 503 is updated into 
the maximum object record value 205 received from the 
server device 102. After then, the flow advances to the 
step S2205. At this time, the object numbers of the newly 
formed object records are rendered one by one at the 

so value obtained by adding "1" to the maximum object 
record value 606 of the data memory device 503 to the 
maximum object record value 205 received from the 
server device 102. 

In the step S2205, a loop variation r is set to be "1 'V 

ss and the flow enters into the object updating loop. Then, 
it is checked in a step S2206 whether or not the object 
record of which object number is r exists in the data 
memory device 503. If not exist, the flow advances to a 
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step S2212. On the other hand, if exists, the flow ad- 
vances to a step S2207. In the step S2207. the attribute 
inquiry comnnand of the object record r is issued to the 
server device 102. In a step S2208, the reply from the 
server device 102 is received. Then, it is judged in a step 
S2209 whether or not the received value is the data rep- 
resenting "deletion". If "deletion", the object record r of 
the data memory device 503 is deleted in a step S221 0, 
and the flow advances to the step S2212. On the other 
hand, if not "deletion", the object record r of the data 
memory device 503 is updated in a step S2211 . and the 
flow advances to the step S2212. 

In the step S2212, the value of r is compared with 
the maximum object record value 606. If the value of r 
is equal to or larger than the maximum object record val- 
ue 606, the flow escapes from the loop and then advanc- 
es to the step S2003 to update the user record of the 
data memory device 503. On the other hand, if the value 
of r is less than the maximum object record value 606, 
" 1 " Is added to the value of r and the flow returns to the 
step S2206. 

When the updating of the object record of the data 
memory device 503 in the step S2002 terminates as 
above, the updating of the user record of the data mem- 
ory device 503 in the step S2003 is performed in the 
same manner. 

Fig. 23 is the flow chart showing the detailed 
processing In the step S2003 to update the user record 
of the data memory device 503. 

In the flowchart of the step S2003, the processing 
is identical with that in the step S2002 explained in Fig. 
22 except for a processing target. Namely, in the step 
S2003, the target is not the object record but is the user 
record. Therefore, detailed explanation thereof will be 
omitted. 

Fig. 24 is the flow chart showing the detailed 
processing in the step S2004 to perform the sketch 
based on the data stored in the data memory device 
503. 

Initially, in a step S2401, the object record is read 
from the data memory device 503. Then, it is judged in 
a step S2402 whether or not reading of all the object 
records terminates. If not terminates, the flow again re- 
turns to the step S2401 to read the next object record. 
On the other hand: if terminates, in a step S2403, the 
object Is arranged on the initial tree according to the po- 
sition data included in the attribute field of the object 
record. 

Subsequently, the same processing as that in the 
steps S2401 to S2403 is performed on the user record. 
Then, in a step S2404, the user record is read from the 
data memory device 503. and it is judged in a step 
S2405 whether or not reading of all the object records 
terminates. If not terminates, the flow again returns to 
the step S2404 to read the next user record. On the oth- 
er hand, if terminates, in a step S2406, the read user 
record is arranged on the tree formed in the step S2403, 
according to the attribute field of the user record. Thus, 



a humanoid model of the another user who shares the 
virtual environment is arranged in the virtual environ- 
ment, whereby it can be confirmed at which position and 
which direction the another user observes. Finally in a 

s step S2407. the tree formed in the step S2406 is 
searched and the sketch processing Is performed. 

In the first embodiment, when the three-dimerision- 
al virtual environment is constructed, like the prior art 
method (1), the elements (coordinate conversion data, 

10 shape data, surface attribute data, illumination and the 
like) composing the virtual environment is expressed by 
means of tree structure. This is because the space, the 
ground, buildings, rooms, fumiture, illuminators, orna- 
ments and the like composing the three-dimensional 

15 space are considered to be essentially in hierarchical 
nested relation. For example, since the ornament on the 
desk Is moved according to arrangement of the desk, 
the ornament is in the relation depending on the ar- 
rangement of the desk. Therefore, often it is better con- 

20 venienl for the ornament to be relatively arranged based 
on the coordinate system in which the desk exists, there- 
by applying the data structure in which the arrange- 
ments of such the elements are in the hierarchically de- 
pendent relation. As the method to do so, the virtual en- 

25 vlronmenl is expressed by n-tree structure. 

Fig. 25 shows an example of one simple virtual en- 
vironment. In this case, if it pays attention to a room, a 
desk and a sofa in the space, the room is described on 
a coordinate system converted from a world coordi- 

30 nate system Cq through a coordinate conversion Tg- and 
the desk and the sofa in this room are described respec- 
tively on coordinate systems C3 and C4 respectively 
converted from the coordinate system C2 through coor- 
dinate conversions T3 and T4. Further a pot on the desk 

35 is described on a coordinate system C5 converted from 
the coordinate system C3 through a coordinate conver- 
sion T5. Unlike the prior art method (1). In the first em- 
bodiment, the object included in the tree structure can 
be described by using not only the geometrical data but 

-^0 also the light space data. In Fig. 25, the light space data 
is arranged on the desk, and this data is described on 
a coordinate system converted from the coordinate 
system C3 through a coordinate conversion Tg. 

In the prior art method (4) wherein the virtual envi- 

•^5 ronment is shared by the plural users, it is impossible to 
share by the plural users the virtual environment which 
Is described by mixing the data such as the light space 
data different in characteristic from the shape data. 
However, In the first embodiment, the light space data 

50 as welt as the shape data is described as one element 
of the identical tree structure, and this tree structure is 
maintained to be identical among the plural users. Thus, 
the virtual environment In which the object having the 
complicated shape is expressed as the light space data 

55 and the data representing the room or the row of houses 
is expressed as the shape data can be shared by the 
plural users. 

Fig. 26 shows the tree structure schematically ex- 
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pressing such the virtual environment shown in Fig. 25. 
Further, Fig. 27 shows the data structure when such the 
tree-structure data is expressed by text data for record- 
ing. In Fig. 27. a keyword "Separator" expresses a top 
of the partial tree, and a keyword "Transform" expresses 
a coordinate conversion. It should be noted that other 
keywords may be used instead of "Separator", "Trans- 
form" and the like if they are appropriately distinguished. 
Further. It should be noted that breakoff symbols such 
as and may be replaced by other symbols or char- 
acter strings. 

In such the virtual environment wherein the light 
space data is described as one element composing the 
tree structure, in the case where the light space data is 
shared by the plural users, by communicating the object 
record which has the light space data and the position 
data as the attribute field and has as the position data 
the data obtained through the coordinate conversion Tg 
from the position In the tree structure on which the light 
space data is arranged, the light space data in the virtual 
environment can be shared by the plural users. 

Fig. 28 is the flow chart showing the detailed 
processing in the step S2407 to search the tree in such 
the virtual environment as expressed by the above tree 
structure and perform the sketch processing. 

In order to cause the observer to observe the virtual 
environment, it is necessary to sketch on the display 
screen how the virtual environment is seen or viewed at 
the observer's viewpoint. In the data tree of n trees 
shown in Fig. 26, all nodes are traced by preferentially 
searching the left node and the depth of the tree struc- 
ture to access the data of all the nodes composing this 
tree. At this time, the sketch is sequentially performed 
on the display screen at the time when accessing the 
node, on the basis of the information written within the 
accessed node. This is the basic method to perform the 
sketch processing. 

Initially, in a step S2801 . setting of the observation 
display screen is obtained. Then, the position and direc- 
tion of the observation viewpoint are obtained from the 
user record in a step S2802, and the searching of the 
tree structure starts in a step S2803. As described 
above, the search processing is performed in the man- 
ner of preferentially searching the left node and the 
depth of the tree structure (step S2804). The node is 
searched until this searching in the step S2804 termi- 
nates (step S2805). 

If the content of the searched node is the coordinate 
conversion (step S2806), the flow advances to current 
coordinate conversion processing in a step S2807. In 
this processing, if there has been no node which is the 
coordinate conversion, the presently existent coordi- 
nate conversion is set to be the current coordinate con- 
version which acts as the basic coordinate conversion 
of the partial tree deeper than the depth of the node be- 
ing searched in the tree. On the other hand, if the node 
which Is the coordinate conversion newly appears, the 
already existent coordinate conversion is multiplied by 



the newly appeared coordinate conversion to update the 
current coordinate conversion. Thus, the updated coor- 
dinate conversion is set to be the current coordinate 
conversion of the partial tree deeper than the depth of 
5 the node In the tree. When the searching Is performed 
in the deeper direction of the tree, by handing the 
present current coordinate conversion, the current co- 
ordinate conversion is provided to the next depth. If 
there is no branch In the deeper direction and thus the 
10 searching is performed in the shallower direction by one 
step, the current coordinate conversion supposed to be 
updated at the deeper state in the tree is not provided 
to the shallower direction. Instead, the current coordi- 
nate conversion which must have been existent when 
IS the tree at that depth was searched before is used. 

If the searched node is not the coordinate conver- 
sion in the step S2806, the flow advances to a step 
S2808. In this step, If the content of the node represents 
the surface attribute, the flow advances to a step S2809 
20 to perform the current surface attribute processing. In 
this processing, the surface attribute is set to be the cur- 
rent surface attribute which acts as the basic surface 
attribute of the partial tree deeper than the depth of the 
node in the tree at the time when such the node appears. 
25 When the searching is performed In the deeper direction 
of the tree, by handing the present current surface at- 
tribute, the current surface attribute is provided to the 
next depth. If there Is no branch In the deeper direction 
and thus the searching is performed in the shallower di- 
30 rection by one step, the current surface attribute sup- 
posed to be updated at the deeper state in the tree is 
not provided to the shallower direction. Instead, the cur- 
rent surface attribute which must have been existent 
when the tree at that depth was searched before is used . 
35 If the searched node does not represent the surface 
attribute In the step 52808, the flow advances to a step 
S2810. If the content of the searched node represents 
a geometrical shape model in the step S2810, the flow 
advances to a step 52811 to perform the geometrical 
40 shape model sketch processing. In this processing, the 
coordinate conversion on position, direction, scale and 
the like of that shape model is performed by using the 
current coordinate conversion, and the sketching is per- 
formed on the display screen provided to the observer 
■^5 as a two-dimensional image by using the current surface 
attribute in the above-described conventional methods 
(1 ). At that time, a depth value storing map correspond- 
ing to each pixel of the provided display screen has been 
previously prepared. When the shape model is sketched 
50 in the sketch processing, the depth value of the three- 
dimensional position on the surface of the shape model 
at the position corresponding to each pixel of the display 
screen viewed from the viewpoint position Is written at 
the same pixel in the depth value storing map. At that 
55 lime, if the depth-value writing to the depth value storing 
map has been performed through such the processing, 
and if the depth value obtained to newly perform the 
sketching Is larger than the already stored depth value. 
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the pixel writing to the provided sketch and the rewriting 
of the depth value storing map are not performed. 

If the searched node does not represent the geo- 
metrical shape model in the step S2810, the flow ad- 
vances to a step S2812. If the content of the searched 
node represents the light space data in the step S2812, 
the flow advances to a step S281 3 wherein the sketch 
processing Is performed from the light space data, tn 
this processing, the coordinate conversion on position, 
direction, scale and the like of the light space data is 
initially performed by using the current coordinate con- 
version. Then, it is calculated at which position and in 
which direction the observation viewpoint position is lo- 
cated in the sight of the standard plane of the light space 
data. Based on the calculated results or the like, a 
sketching rule in forming the image which is to be formed 
from the light space data in such observation conditions 
is determined, and the image to be displayed on the pro- 
vided display screen is actually formed. At that time, like 
the sketching from the geometrical shape model, the 
depth is judged in the unit of pixel on the display screen. 
That is. the distance between the viewpoint position and 
the position at which the standard plane of the light 
space data is located is considered as the depth value 
of the light space data, and the obtained value is com- 
pared with the depth value stored in the depth value stor- 
ing map. On the pixel of which depth value is smaller 
than the value of the depth value storing map. the image 
formed from the light space is overwritten to update the 
depth value storing map. On the other hand, if not small- 
er, no processing is performed. 

If the searched node does not represent the light 
space data in the step S2812, or if the processing in the 
steps S2807, S2809. S2811 and S2813 terminate, the 
flow again returns to the step S2804 to perform the 
search processing on the tree. Then, if it is judged in the 
step S2805 that the searching on the tree terminated, 
the sketch processing terminates. 

When the sketching based on the updated object 
record and the updated user record terminates, the flow 
advances to the step S2005 to perform the processing 
according to the user's instruction. In this processing, 
the user can instruct the seven kinds of processing, i.e.: 
to add the object, to delete the object, to lock the object, 
to unlock the object, to change the attribute of the object, 
to change the attribute of the user, and to terminate or 
end the processing. 

Fig. 33 is the flow chart showing the detailed 
processing in the step S2005 to issue the desired com- 
mand according to the user's instruction. 

If the user instructs to add the object, in a step 
S3301, the attribute ol the object previously prepared 
by the user is added to the object record in the com- 
mand. The attribute of the object includes the geomet- 
rical shape data having the surface attribute data, or the 
light space data. Then, in a step S3302, the object 
record addition command is issued to the server device 
102* When the command issuing terminates, the flow 



returns to the step S2002 to again start the sketch loop. 

When the user instructs to delete the object R. in a 
step S3303, the object record R deletion command is 
issued to the server device 1 02 by the terminal equip- 

5 ment, and the flow returns to the step S2002, 

Similarly, the object record R lock command is is- 
sued in a step S3304 when the user instructs to lock the 
object R, or the object record R unlock command is is- 
sued in a step S3305 when the user instructs to unlock 

10 the object R. Then, the flow returns to the step S2002. 
When the user instructs to change the attribute of 
the object, the changing corresponding to the operation 
by the user is performed on the object record (step 
S3306). and the object record change command includ- 

^5 ing the changed object record is issued (step S3307). 

When the user performs the operation such as 
movement of the viewpoint which causes some change 
in the user attribute, the change corresponding to such 
the operation is added to the user record in a step 

20 S3308, and then the user record change command is 
issued in a step S3309. 

When the user instructs to terminate the process- 
ing, the user record deletion command corresponding 
to such the user is Issued to the server device 102 by 

25 the terminal equipment, and the processing terminates. 
It is possible that, according to the data contents, 
these commands which change the data in the data 
memory device 101 connected to the server device 102 
can not be accepted by the server device 102. For ex- 

30 ample, even if the user instructs to delete the object 
record not locked, the sen/er device 102 does not delete 
such the object record. However, it is unnecessary for 
the terminal equipment to confirm whether or not the 
command issued by the equipment itself has been ac- 

35 cepted by the server device 102. That is, the terminal 
equipment can start the next processing immediately af- 
ter issuing the command. Further, these commands are 
sent to all the other terminal equipments through the 
server device 1 02, and thus the data in the data memory 

40 device 503 within each equipment is changed. 

Figs. 29 and 30 respectively show examples of 
sketched images on the display screens when one vir- 
tual environment is shared by users 1 and 2. In Fig. 29. 
the virtual environment corresponding to the viewpoint 
position and direction of the user 1 is displayed on the 
display screen of the terminal equipment of the user 1. 
In Fig. 30, the virtual environment corresponding to the 
viewpoint position and direction of the user 2 is dis- 
played on the display screen of the terminal equipment 

so of the user 2. 

In Figs. 29 and 30, the respective users observe the 
identical virtual environment at the different positions 
and directions. In the drawings, it should be noted that 
a stuffed rabbit is described by the light space data, and 

55 the room and a humanoid model representing the user 
are described by the shape models. In Fig. 29 showing 
the virtual environment viewed by the user 1 . the hu- 
manoid model of the user 2 is displayed at the center of 
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the display screen to represent that the user 2 stands in 
front of the user 1. Further geometrical objects acting 
as marks or signs are displayed on the periphery of the 
stuffed rabbit to represent that the user 1 is locking the 
rabbit. At that time^ in Fig. 30 showing the virtual envi- 
ronnnent viewed by the user 2, the humanoid model of 
the user 1 is displayed at the center of the display screen 
to represent that the user 1 stands in front of the user 
2. Further, geometrical objects having an inherent color 
of the user 1 are displayed above and below the stuffed 
rabbit to represent that the user 1 is locking the rabbit. 

Then, if the user 1 instructs the rabbit object locked 
in the virtual environment to change its position, the 
command is sent from the terminal equipment of the us- 
er 1 to the server device 102 and further sent to the ter- 
minal equipment of the user 2. Therefore, the example 
of the sketched image on the display screen shown in 
Fig. 29 is changed into that shown in Fig. 31. Further, 
according to movement of the rabbit object by the user 
1 , the example of the sketched image on the display 
screen shown in Fig. 30 is changed into that shown in 
Fig. 32. 

(Second Embodiment) 

In the first embodiment, the image to be displayed 
is formed from the light space data. However in the 
present embodiment, an image necessary for display is 
formed from a multi-viewpoint image instead of the light 
space data by using the following method. 

Fig. 38 is a view showing a principle that the image 
is restructured from the multi-viewpoint image inputted. 
As shown in the drawing, it is necessary as a photo- 
graphing condition of the multi-viewpoint image to pre- 
viously arrange cameras on a straight line perpendicular 
to a photographing direction and photograph the image 
by these cameras. Reference numeral 3801 denotes an 
object to be photographed, reference numeral 3602 de- 
notes photographing viewpoints arranging line obtained 
by linking photographing points of the input image, ref- 
erence numeral 3803 denotes a virtual CCD plane of a 
virtual camera, and reference numeral 3804 denotes a 
virtual camera put at an observer's position (x\ z"). 

The processing to form the image from the multi- 
viewpoint image will be explained with reference to a 
flow chart shown in Fig. 39. Initially, in a step S3901 , a 
target line j is set to a head line of an image R and a 
target pixel is set to a left-end pixel of the line j. Then, 
in a step S3902, a position of an image Q on the photo- 
graphing viewpoints arranging line 3802 corresponding 
to an i-th pixel of the line j of the image P is calculated 
as follows. 

That is, it is assumed that one point A in the object 
is projected at a pixel position Pi of the virtual camera 
at the viewpoint position R and that a position at which 
a line linking the positions A and P crosses the photo- 
graphing viewpoints arranging line 3802 is Q. At this 
time, it can be understood from Fig. 38 that the photo- 



graphed object at the pixel position Pi is equivalent to 
that at a pixel position Qi in the image photographed at 
the viewpoint position Q. According to a geometrically 
restricted condition in Fig. 38, the viewpoint position A 

5 in the X coordinate can be expressed by the equation 
(1 ). It is assumed that a central pixel position on a scan- 
ning line is at the 0-th pixel, a pixel pitch and a focusing 
distance of the virtual camera 3804 are respectively d 
and f, and a distance between an origin and the photo- 

10 graphing viewpoints arranging line 3802 is g. 

X = X' + i-d.(g-2')/f (3) 

IS Similarly, the photographed object at a pixel posi- 
tion Pj of the virtual camera at the viewpoint position P 
is equivalent to that at a pixel position Rj of the image 
at a viewpoint position R stored in the memory device. 
According to such the method, for example, when the 
20 image is reconstructed by copying the i-th pixel value 
on the line j of the image Q to the i-th pixel on the line j 
of the image P, the obtained image is distorted. Namely 
it is possible that the object in the image expands or 
shrinks longitudinally. Therefore, in order to eliminate 
25 such inconvenience, a line number n of the image Q cor- 
responding to the line j of the image P is obtained in a 
step S3903. A method to obtain the line number n of the 
image Q will be explained with reference to Fig. 40. 
Fig. 40 is a view showing a principle to correct dis- 
00 tortion in the image which is restructured as above. In 
the drawing, reference numeral 4001 denotes an object 
to be photographed, reference numeral 4002 denotes 
an image at a viewpoint position P intended to be re- 
structured, and reference numeral 4003 denotes an im- 
35 age at a viewpoint position step S on the photographing 
viewpoints arranging line 3802. 

With respect to one point B in the object 4001 , it is 
assumed that the point B is close to the Y axis, a Z co- 
ordinate value Pz of the image 4002 at the viewpoint 
•to position P intended to be restructured and a Z coordi- 
nate value Sz of the image 4003 at the viewpoint posi- 
tion step S on the photographing viewpoints arranging 
line 3802 are sufficiently large, or the Z coordinate value 
Pz of the image 4002 at the viewpoint position P intend- 
•f5 ed to be restructured and the Z coordinate value Sz of 
the image 4003 at the viewpoint position step S on the 
photographing viewpoints arranging line 3802 are sub- 
stantially identical. At that time, light from the point B Is 
recorded on a line m in the image 4002 at the viewpoint 
so position P intended to be restructured and a line n in the 
image 4003 at the viewpoint position step S on the pho- 
tographing viewpoints arranging line 3802. Therefore, if 
it is assumed that a pixel pitch of the virtual CCD 3803 
is d, the focusing distance of the virtual camera 3804 is 
ss i. and the number of lines of the virtual CCD 3803 is N, 
following equations are obtained. 
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Pz-lana = SzlanP (4) 



tana = d.(N/2-m)/f (5) 



tanp = d.(N/2-n)/f (6) 

Further by using the equations (4). (5) and (6). a io 
following equation is obtained. 

n = N/2 + (m - N/2).Sz/Pz (7) 

15 

Therefore^ tlie value of the line m in the image 4002 
at the viewpoint position P intended to be restructured 
is equivalent to the value (obtained by equation (7)) of 
the line n in the image 4003 at the viewpoint position 
step S on the photographing viewpoints arranging line 20 
3802- 

Thus, after the processing in the step 53903= the 
flow advances to a step S3904 to copy the value of the 
i-th pixel of the line n of the image Q to the i-th pixel of 
the line j of the image P. By such the processing, the 2S 
phenomenon that the image is distorted when the image 
at the viewpoint position out of the photographing view- 
points arranging line 3802 is restructured can be sup- 
pressed to some extent. 

Then, the flow advances to a step S3905 to judge 30 
whether or not the processing on all the pixels in the 
target line j has terminated If terminated, the flow ad- 
vances to a step S3907. On the other hand, if not termi- 
nated, the flow advances to a step S3906. In the step 
S3906. the target pixel i is shifted to a rightHnext pixel, 35 
and the flow returns to the step S3902. 

Further it is judged in the step S3907 whether or 
not the processing on all the lines in the image P has 
terminated. If terminated, this subroutine terminates. On 
the other hand, if not terminated, the flow advances to 40 
a step S3908 to shift the target line j to the next line and 
set the target pixel i to the left end of the line j, and then 
the flow returns to the step S3902. 

As above, in the case where the group of the multi- 
viewpoint images photographed on the photographing -*5 
viewpoints arranging line 3802 at minute intervals has 
been obtained, if the same processing is repeated on 
all the scanning lines on the basis of the above-de- 
scribed concept, the Image at the viewpoint position out 
of the photographing viewpoints arranging line 3802 can 
be restructured. 

(Third Embodiment) 

In the first embodiment, as shown in Fig. 1 , the vir- 
tual environment is shared by the plural terminal equip- 
ments by wholly changing the data representing the 
state* of the virtual environment through the server de- 



vice 102. However the same effect can be derived even 
by using a such communication arrangement not using 
the server device 102 as shown in Fig. 41. 

In Fig. 41. terminal equipments 4101 to 4104 are 
network-connected by using a bus 4105. Each terminal 
equipment performs the same processing as that per- 
formed by the terminal equipment in the first embodi- 
ment. However it is necessary in the present embodi- 
ment that the command issued to the server device 102 
In the first embodiment is set to be sent to ail the terminal 
equipments connected in the network. In addition, it is 
necessary In the present embodiment that the terminal 
equipment is set to receive all the commands issued by 
the other terminal equipments and perform the same 
processing as that performed by the server device 102 
in the first embodiment on the data within its own data 
memory device 503. 

Further, In the first embodiment, when the data 
memory device 101 on the server device 102 side and 
the data memory devices 503 included in the plural ter- 
minal equipments are made physically common by us- 
ing some means, a virtual space can be shared by the 
plural users even though the communication between 
the terminal equipment and the server device 102 as in 
the first embodiment is not performed. 

Similarly, in the third embodiment, when the data 
memory devices 503 respectively included in the plural 
terminal equipments are made physically common by 
using some means, the virtual space can be shared by 
the plural users even though the communication be- 
tween the terminal equipments as in the third embodi- 
ment is not performed. 

As explained above in detail, according to the 
present embodiments, it is possible to derive the effect 
that the virtual environment including not only the simple 
shape data but also the highly complicated shape data 
can be shared by the plural users. 

Further in such the shared virtual environment, it is 
possible to derive the effect that, when the user per- 
forms some processing on some object, the other users 
can easily view and recognize the change in the virtual 
environment caused by such the processing. 

Claims 

1. An image processing apparatus comprising: 

tree-structure data memory means for storing 
tree-structure data, the tree-structure data in- 
cluding as components shape data geometri- 
cally describing a three-dimensional space and 
a three-dimensional object and light space data 
describing the three-dimensional space and 
the three-dimensional object based on a real- 
photographed image: 

displaying means for displaying a virtual three- 
dimensional space in which the shape data and 
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the light space data are mixed with each other 
based on the tree-structure data stored in said 
tree-structure data memory means: 
processing means for processing at least one 
of the shape data and the light space data both s 
displayed; 

changing means for changing the tree-struc- 
ture data stored in said tree-structure data 
memory means, based on the processing con- 
tent by said processing means: 
re-displaying means for re-displaying the virtu- 
al three-dimensional space, based on the data 
changing content by said changing means; 
indication information generating means for 
generating indication information indicating the ^5 
changing content of the tree-structure data, 
based on the processing content by said 
processing means; and 
transmitting means for transmitting the indica- 
tion information generated by said indication in- 
formation generating means to an another in- 
formation processing apparatus connected to 
said image processing apparatus. 

2. An apparatus according to Claim 1 , wherein said ^5 
processing means moves, adds, deletes or chang- 
es at least one of the shape data and the light space 
data. 

3. An apparatus according to Claim 1 , wherein the an- 30 
other information processing apparatus is a data 
distributing apparatus for distributing data to plural 
image processing apparatuses. 

4. An apparatus according to Claim 1 , wherein the an- 3S 
other information processing apparatus is an image 
processing apparatus having the same function as 
that of said image processing apparatus. 

5. An apparatus according to Claim 1 , further compris- 
ing: 

position and direction data memory means for 
storing data representing a viewpoint position 
and a sight-line direction of a user of said image 
processing apparatus in the virtual three-di- 
mensional space, and 

wherein said displaying means displays the vir- 
tual three-dimensional space based on the data 
representing the viewpoint position and the so 
sight-line direction stored in said position and 
direction data memory means. 

6. An apparatus according to Claim 5= wherein said 
transmitting means transmits the indication infor- 55 
mation generated by said indication information 
generating means, and also transmits the data rep- 
resenting the viewpoint position and the sight-line 



direction stored in said position and direction data 
memory means. 

7. An apparatus according to Claim 1 , wherein three- 
dimensional innage data led from two-dimensional 
multi-viewpoint images is used instead of the light 
space data. 

8. An apparatus according to Claim 1 , wherein, in the 
processing by said processing means, said re-dis- 
playing means discriminatingly displays the shape 
data or the light space data being the object of the 
processing. 

9. An image processing apparatus comprising: 

tree-structure data memory means for storing 
tree-structure data, the tree-structure data in- 
cluding as components shape data geometri- 
cally describing a three-dimensional space and 
a three-dimensional object and light space data 
describing the three-dimensional space and 
the three-dimensional object based on a real- 
photographed image: 

displaying means for displaying a virtual three- 
dimensional space in which the shape data and 
the light space data are mixed with each other, 
based on the tree-structure data stored in said 
tree-structure data memory means; 
' receiving means for receiving indication infor- 
mation indicating a changing content of the 
tree-structure data, transmitted from an anoth- 
er information processing apparatus connected 
to said image processing apparatus; 
changing means for changing the tree-struc- 
ture data stored in said tree-structure data 
memory means, based on the indication infor- 
mation received by said receiving means; and 
re-displaying means for re-displaying the virtu- 
al three-dimensional space, based on the data 
changing content by said changing means. 

10. An apparatus according to Claim 9, wherein the 
changing content indicated by the indication infor- 
mation received by said receiving means is to 
move, add, delete or change at least one of the 
shape data and the light space data. 

11. An apparatus according to Claim 9, wherein the an- 
other information processing apparatus is a data 
distributing apparatus for distributing data to plural 
image processing apparatuses. 

12. An apparatus according to Claim 9, wherein the an- 
other information processing apparatus is an image 
processing apparatus having the same function as 
that of said image processing apparatus. 
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13. An apparatus according to Claim 9. wherein said 
receiving nneans receives the indication infornnation 
indicating the changing content of the tree-structure 
data, and also receives data representing a view- 
point position and a sight-line direction of a user of 
the another image processing apparatus in the vir- 
tual three-dimensional space. 

14. An apparatus according to Claim 13, further com- 
prising data generating means for generating data 
representing a virtual model of the user of the an- 
other image processing apparatus in the virtual 
three-dimensional space, based on the data repre- 
senting the viewpoint position and the sight-line di- 
rection received by said receiving means, and 

wherein said re-displaying means displays 
the virtual model of the user of the another image 
processing apparatus, in the virtual three-dimen- 
sional space. 

15. An apparatus according to Claim 9, wherein three- 
dimensional image data led from two-dimensional 
multi-viewpoint images is used instead of the light 
space data. 

1 6. An apparatus according to Claim 9, wherein said re- 
displaying means discriminatingly displays the 
shape data or the light space data being the object 
of the processing, based on the indication informa- 
tion received by said receiving means and indicat- 
ing the changing content in the tree-structure data. 

17. An image processing method which is used in an 
image processing apparatus comprising a tree- 
structure data memory means for storing tree-struc- 
ture data, the tree-structure data including as com- 
ponents shape data geometrically describing a 
three-dimensional space and a three-dimensional 
object and light space data describing the three-di- 
mensional space and the three-dimensional object 
based on a real-photographed image, said method 
comprising: 

a displaying step of displaying a virtual three- 
dimensional space in which the shape data and 
the light space data are mixed with each other 
based on the tree-structure data stored in the 
tree-structure data memory means: 
a processing step of processing at least one of 
the shape data and the light space data both 
displayed: 

a changing step of changing the tree-structure 
data stored in the tree-structure data memory 
means., based on the processing content in said 
processing step: 

a re-displaying step of re-displaying the virtual 
three-dimensional space, based on the data 
changing content in said changing step: 



an indication information generating step of 
generating indication information Indicating the 
changing content of the tree-structure data, 
based on the processing content in said 

5 processing step; and 

a transmitting step of transmitting the indication 
information generated in said indication infor- 
mation generating step to an another informa- 
tion processing apparatus connected to the im- 

10 age processing apparatus. 

18. A method according to Claim 17, wherein in said 
processing step at least one of the shape data and 
the light space data is moved, added, deleted or 

'5 changed. 

19. A method according to Claim 17, wherein the an- 
other information processing apparatus is a data 
distributing apparatus for distributing data to plural 

20 image processing apparatuses. 

20. A method according to Claim 17, wherein the an- 
other information processing apparatus is an image 
processing apparatus having the same function as 

2S that of the above-described image processing ap- 
paratus. 

21. A method according to Claim 17, wherein the image 
processing apparatus further comprises a position 

30 and direction data memory means for storing data 
representing a viewpoint position and a sight-tine 
direction of a user of the image processing appara- 
tus In the virtual three-dimensional space, and 

in said displaying step the virtual three-dimen- 

3S sional space is displayed based on the data repre- 
senting the viewpoint position and the sight-line di- 
rection stored in the position and direction data 
memory means. 

-to 22. A method according to Claim 21, wherein, in said 
transmitting step, the indication information gener- 
ated in said indication information generating step 
is transmitted, and also the data representing the 
viewpoint position and the sight-line direction stored 
in the position and direction data memory means is 
transmitted. 

23. A method according to Claim 17, wherein three-di- 
mensional image data led from two-dimensional 

so multi-viewpoint images is used instead of the light 
space data. 

24. A method according to Claim 17. wherein, in said 
re-displaying step, the shape data or the light space 

ss data being the object of the processing in said 
processing step is discriminatingly displayed. 

25. An image processing method which is used in an 
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image processing apparatus comprising a tree- 
structure data memory means for storing tree-struc- 
ture data, the tree-structure data including as com- 
ponents shape data geometrically describing a 
three-dimensional space and a three-dimensional 
object and light space data describing the three-di- 
mensional space and the three-dimensional object 
based on a real-photographed image, said method 
comprising: 

a displaying step of displaying a virtual three- 
dimensional space in which the shape data and 
the light space data are mixed with each other, 
based on the tree-structure data stored in the 
tree-structure data memory means; 
a receiving step of receiving indication informa- 
tion indicating a changing content of the tree- 
structure data, transmitted from an another in- 
formation processing apparatus connected to 
the image processing apparatus: 
a changing step of changing the tree-structure 
data stored in the tree-structure data memory 
means, based on the indication information re- 
ceived in said receiving step: and 
a re-displaying step of re-displaying the virtual 
three-dimensional space, based on the data 
changing content in said changing step. 

26. A method according to Claim 25. wherein the 
changing content indicated by the indication infor- 
mation received in said receiving step is to move, 
add, delete or change at least one of the shape data 
and the light space data. 

27. A method according to Claim 25, wherein the an- 
other information processing apparatus is a data 
distributing apparatus for distributing data to plural 
image processing apparatuses. 

28. A method according to Claim 25, wherein the an- 
other information processing apparatus is an image 
processing apparatus having the same function as 
that of the above-described image processing ap- 
paratus. 

29. A method according to Claim 25, wherein, in said 
receiving step, the indication information indicating 
the changing content of the tree-structure data is 

received, and also data representing a viewpoint 
position and a sight-line direction of a user of the 
another image processing apparatus in the virtual 
three-dimensional space is received. 

30. A method according to Claim 29. further comprising 
a data generating step of generating data repre- 
senting a virtual model of the user of the another 

image processing apparatus in the virtual three-di- 
mensional space, based on the data representing 



the viewpoint position and the sight-line direction re- 
ceived in said receiving step, and 

wherein in said re-displaying step the virtual 
model of the user of the another image processing 
s apparatus is displayed in the virtual three-dimen- 
sional space. 

31. A method according to Claim 25. wherein three-di- 
mensional image data led from two-dimensional 

10 multi-viewpoint images is used instead of the light 
space data. 

32. A method according to Claim 25, wherein in said re- 
displaying step the shape data or the light space 

IS data being the object of the changing in said chang- 
ing step is discriminatingly displayed based on the 
indication information received in said receiving 
step and indicating the changing content in the tree- 
structure data. 

20 

33. A computer-readable memory medium which is 
used in an image processing apparatus comprising 
a tree-structure data memory means for storing 
tree-structure data, the tree-structure data including 

2S as components shape data geometrically describ- 
ing a three-dimensional space and a three-dimen- 
sional object and light space data describing the 
three-dimensional space and the three-dimension- 
al object based on a real-photographed image, and 

30 which stores a computer program including: 

a displaying module for displaying a virtual 
three-dimensional space in which the shape 
data and the light space data are mixed with 
35 each other based on the tree-structure data 

stored in the tree-structure data memory 
means; 

a processing module for processing at least 
one of the shape data and the light space data 

-/o both displayed: 

a changing module for changing the tree-struc- 
ture data stored in the tree -structure data mem- 
ory means, based on the processing content; 
a re-displaying module for re-displaying the vir- 

45 tual three-dimensional space, based on the da- 

ta changing content: 

an indication information generating module for 
generating indication information indicating the 
changing content of the tree-structure data. 
so based on the processing content: and 

a transmitting module for transmitting the gen- 
erated indication information to an another in- 
formation processing apparatus connected to 
the image processing apparatus. 

55 

34. A computer-readable memory medium which is 
used in an image processing apparatus comprising 
a tree-structure data memory means for storing 
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tree-structure data, the tree-structure data including 
as components shape data geometrically describ- 
ing a three-dimensional space and a three-dimen- 
sional object and light space data describing the 
three-dimensional space and the three-dimension- s 
al object based on a real-photographed image, and 
which stores a computer program including: 

a displaying module for displaying a virtual 
three-dimensional space in which the shape io 
data and the light space data are mixed with 
each other, based on the tree-structure data 
stored in the tree-structure data memory 
means: 

a receiving module for receiving indication in- ^5 
formation indicating a changing content of the 
tree-structure data, transmitted from an anoth- 
er information processing apparatus connected 
to the image processing apparatus; 
a changing module for changing the tree-struc- 20 
ture data stored in.the tree-structure data mem- 
ory means, based on the received indication in- 
formation: and 

a re-displaying module for re-displaying the vir- 
tual three-dimensional space, based on the da- 2S 
ta changing content. 

35. An image processing system in which a first termi- 
nal equipment is connected to a second terminal 
equipment through a server comprising: 20 

said first terminal equipment further compris- 
ing. 

first tree-structure data memory means for 3S 
storing tree-structure data, the tree-struc- 
ture data including as components shape 
data geometrically describing a three-di- 
mensional space and a three-dimensional 
object and light space data describing the -^0 
three-dimensional space and the three-di- 
mensional object based on a real-photo- 
graphed image, 

first displaying means for displaying a vir- 
tual three-dimensional space in which the -^s 
shape data and the light space data are 
mixed with each other, based on the tree- 
structure data stored in said first tree-struc- 
ture data memory means, 
processing means for processing at least so 
one of the shape data and the light space 
data both displayed, 

first changing means for changing the tree- 
structure data stored in said first tree-struc- 
ture data memory means, based on the 55 
processing content by said processing 
means, 

first re-displaying means for re-displaying 



the virtual three-dimensional space, based 
on the data changing content by said first 
changing means. 

indication information generating means 
for generating indication information Indi- 
cating the changing content of the tree- 
structure data, based on the processing 
content by said processing means, and 
first transmitting means for transmitting the 
indication information generated by said in- 
dication information generating means to 
said server connected to said first terminal 
equipment: 

said server further comprising, 

first receiving means for receiving the indi- 
cation information transmitted from said 
first terminal equipment, and 
second transmitting means for transmitting 
the indication information received by said 
first receiving means to said second termi- 
nal equipment: and 

said second terminal equipment further com- 
prising, 

second tree-structure data memory means 
for storing tree-structure data, the tree- 
structure data including as components 
shape data geometrically describing a 
three-dimensional space and a three-di- 
mensional object and light space data de- 
scribing the three-dimensional space and 
the three-dimensional object based on a 
real-photographed image, 
second displaying means for displaying a 
virtual three-dimensional space in which 
the shape data and the light space data are 
mixed with each other based on the tree- 
structure data stored in said second tree- 
structure data memory means, 
second receiving means for receiving the 
indication information indicating the chang- 
ing content of the tree-structure data, 
transmitted from said server connected to 
said second terminal equipment, 
second changing means for changing the 
tree-structure data stored in said second 
tree-structure data memory means, based 
on the indication Information received by 
said second receiving means, and 
second re-displaying means for re-display- 
ing the virtual three-dimensional space, 
based on the data changing content by 
said second changing means. 

36. An image processing system in which a first termi- 



19 



37 



EP 0 865 001 A2 



38 



nal equipment and a second terminal equipment are 
connected to each other comprising: 

said first terminal equipment further compris- 
ing, 

first tree-structure data memory means for 
storing tree-structure data, the tree-struc- 
ture data including as components shape 
data geometrically describing a three-di- 
mensional space and a three-dimensional 
object and light space data describing the 
three-dimensional space and the three-di- 
mensional object based on a real-photo- 
graphed image, 

first displaying means for displaying a vir- 
tual three-dimensional space in which the 
shape data and the light space data are 
mixed with each other, based on the tree- 
structure data stored in said first tree-struc- 
ture data memory means, 
processing means for processing at least 
one of the shape data and the light space 
data both displayed. 

first changing means for changing the tree- 
structure data stored in said first tree-struc- 
ture data memory means, based on the 
processing content by said processing 
means- 

first re-displaying means for re-displaying 
the virtual three-dimensional space, based 
on the data changing content by said first 
changing means, 

indication information generating means 
for generating indication information indi- 
cating the changing content of the tree- 
structure data, based on the processing 
content by said processing means, and 
transmitting means for transmitting the in- 
dication information generated by said in- 
dication information generating means to 
said second terminal equipment connect- 
ed to said first terminal equipment: and 

said second terminal equipment further com- 
prising, 

second tree-structure data memory means 
for storing tree-structure data, the tree- 
structure data including as components 
shape data geometrically describing a 
three-dimensional space and a three-di- 
mensional object and light space data de- 
scribing the three-dimensional space and 
the three-dimensional object based on a 
real-photographed image, 
second displaying means for displaying a 
virtual three-dimensional space in which 



the shape data and the light space data are 
mixed with each other, based on the tree- 
structure data stored in said second tree- 
structure data memory means, 

5 receiving means for receiving the indica- 

tion information indicating the changing 
content of the tree-structure data, transmit- 
ted from said first terminal equipment con- 
nected to said second terminal equipment, 

10 second changing means for changing the 

tree-structure data stored in said second 
tree-structure data memory means, based 
on the indication information received by 
said receiving means, and 

IS second re-displaying means for re-display- 

ing the virtual three-dimensional space, 
based on the data changing content by 
said second changing means. 



20 



25 



37. A computer-graphics system comprising first and 
second interconnected processing apparatus, each 
having: 



a memory storing shape data defining the sur- 
face of a first three-dimensional object, light 
space data defining a second three-dimension- 
al object, and data defining a hierarchical tree 
structure between the first and second objects: 
display means for displaying images of the ob- 
30 jects in accordance with the stored data; and 

processing means for amending the stored da- 
ta: 

wherein information defining amendments to 
the data in the first apparatus is transmitted to 
35 the second apparatus to cause corresponding 

changes therein. 

38. A system according to claim 37. wherein the first 
and second processing apparatus are interconnect- 

40 ed via a server, the server accessing a memory stor- 
ing data corresponding to the data stored in the first 
and second apparatus. 

39. Processing apparatus for use in a system according 
45 to claim 37 or 38, comprising: 

memory means for storing shape data defining 
the surface of a first three-dimensional object, 
light space data defining a second three-dimen- 
50 sional object, and data defining a hierarchical 

tree structure between the first and second ob- 
jects: 

means for generating image data defining an 
image in accordance with data stored in the 
55 memory means: 

processing means for amending data stored in 

the memory means; and 

means for transmitting data defining amend- 
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merits made by the processing means. 

40. Processing apparatus for i^e in a system according 
to claim 37 or 38, comprising: 

memory means for storing shape data defining 
the surface of a first three-dimensional object, 
light space data defining a second three-dimen- 
sional object, and data defining a hierarchical 
tree structure between the first and second ob- 
jects; 

means for generating image data defining an 
image in accordance with data stored in the 
memory means: 

means for receiving data defining amendments 
to be made to the data; and 
means for amending the data stored in the 
memory means in accordance with the re- 
ceived data. 

41. A computer graphics system comprising: 

memory means storing shape data defining the 
surface of a first three-dimensional object, light 
space data defining a second three-dimension- 
al object and data defining a hierarchical tree 
structure between the first and second objects; 
first processing apparatus connected to the 
memory means, having display means for dis- 
playing images of the objects in accordance 
with the stored data, and means for amending 
the stored data: and 

second processing apparatus connected to the 
memory means, having display means for dis- 
playing images of the objects in accordance 
with the stored data, and means for amending 

the stored data: 

wherein information defining amendments to 
the stored data made by the first apparatus is 
transmitted to the second apparatus. 

42. Apparatus for use in a system according to claim 
41 , comprising: 
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means for receiving object data comprising 
shape data defining the surface of a first three- 
dimensional object, light space data defining a 
second three-dimensional object, and data de- 
fining a hierarchical tree structure between the 
first and second objects: 
means for generating image data defining an 
image in accordance with the received data; 
means for receiving data defining amendments 
to the object data; and 

means for generating amended image data in 
accordance with the received amendment da- 
ta. 

44. A storage device storing computer-readable in- 
structions for causing a programmable processing 
apparatus to have the processing functions of an 
apparatus according to any of claims 39, 40, 42 and 
43. 

45. A signal conveying instructions for causing a pro- 
grammable processing apparatus to have the 
processing functions of an apparatus according to 
any of claims 39. 40, 42 and 43. 
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means for receiving shape data defining the -^5 
surface of a first three-dimensional object, light 
space data defin ing a second three-dimension- 
al object, and data defining a hierarchical tree 
structure between the first and second objects: 
means for generating image data defining an so 
image in accordance with the received data: 
processing means for amending the data: and 
means for transmitting data defining amend- 
ments made by the processing means. 

55 

43. Apparatus for use in a system according to claim 
41 . comprising: 
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